Design and validation of a new Healthcare Systems Usability Scale (HSUS) for clinical decision support systems: a mixed-methods approach

Objective To develop and validate a questionnaire to assess the usability of clinical decision support systems (CDSS) and to assist in the early identification of usability issues that may impact patient safety and quality of care. Design Mixed research methods were used to develop and validate the questionnaire. The qualitative study involved scale item development, content and face validity. Pilot testing established construct validity using factor analysis and facilitated estimates for reliability and internal consistency using the Cronbach’s alpha coefficient. Setting Two hospitals within a single National Health Service Trust. Participants We recruited a panel of 7 experts in usability and questionnaire writing for health purposes to test content validity; 10 participants to assess face validity and 78 participants for the pilot testing. To be eligible for this last phase, participants needed to be health professionals with at least 3 months experience using the local hospital electronic patient record system. Results Feedback from the face and content validity phases contributed to the development and improvement of scale items. The final Healthcare Systems Usability Scale (HSUS) proved quick to complete, easy to understand and was mostly worded by potential users. Exploratory analysis revealed four factors related to patient safety, task execution, alerts or recommendations accuracy, the effects of the system on workflow and ease of system use. These separate into four subscales: patient safety and decision effectiveness (seven items), workflow integration (six items), work effectiveness (five items) and user control (four items). These factors affect the quality of care and clinician’s ability to make informed and timely decisions when using CDSS. The HSUS has a very good reliability with global Cronbach’s alpha 0.914 and between 0.702 and 0.926 for the four subscales. Conclusion The HSUS is a valid and reliable tool for usability testing of CDSS and early identification of usability issues that may cause medical adverse events.


GENERAL COMMENTS
Thank you for inviting me to review this manuscript. In brief, the paper presents the design and validation of a new Healthcare Systems Usability Scale using mixed research methods. It describes some very interesting findings that can be very useful for early identification of usability issues that may impact patient safety and quality of care. There is a critical need for usability scale that takes into consideration different usability aspects in medical context which supports the importance of this work. My impression is that the method proposed here may have scientific potential. I also believe that the subject may matter of relevance to BMJ Open journal.
Below, a list of minor modifications to be addressed by the coauthors: • Overall, I found the findings very interesting and I think this is a nice piece of work to inform further technological development in this area.

VERSION 1 -AUTHOR RESPONSE
Responses to reviewer 1 comments: 1. It describes some very interesting findings that can be very useful for early identification of usability issues that may impact patient safety and quality of care. There is a critical need for usability scale that takes into consideration different usability aspects in medical context which supports the importance of this work. My impression is that the method proposed here may have scientific potential. I also believe that the subject may matter of relevance to BMJ Open journal. Thank you, we are glad that this reviewer found that our study provided interesting findings that can inform smart home development. We had had hoped that our paper would provide this to readers and we are pleased.  Table 2. We have made appropriate correction to the text of the revised version, "Face validity" section (Page 11) and "Content validity" section (Page 15).
6. Page 24, line 11: The reference number 60, is it for how the inclusion of an item in a construct is determined or it is just related to factor loadings? The rest of the claims in this paragraph on this page should be supported by references. Thank you. For more clarity, the reference was replaced at the end of the sentence. 7. The authors should start the discussion by emphasizing on the importance of HSUS as there is need for a reliable and accurate usability scale for health information systems. This would make the discussion stronger. I suggest starting the discussion by line 9 page 29 to line 44, page 30. Then the authors can talk about the methodology or the validation process of the scale. Thank you for this suggestion, the discussion was updated as suggested. 8. In the conclusion, line 52: Is HSUS designed to evaluate just the newly designed systems? In the discussion, the authors have suggested to evaluate the systems after certain time of use. Thank you, a clarification was added to the conclusion, page 34. 9. The authors may perhaps want to summarise the different subscales in the discussion Thank you for this suggestion, HSUS subscales summary was added on page 31. 10. Minor points to be addressed: a. Line 31 needs editing. b. The justification line of Table 1 Table reference error Thank you and we are sorry about these errors. We have revised the text and made the edits in light of the suggestions. We are grateful for the attention to detail that this reviewer has provided.
11. Overall, I found the findings very interesting and I think this is a nice piece of work to inform further technological development in this area. Thank you, we are glad that this reviewer found that our study provided interesting findings that can inform further technological development. We had had hoped that our paper would provide this to readers and we are pleased.
Responses to reviewer 2 comments: 1. The authors have done a good study however the mixed methods study design is not detailed as per mixed method study designs. The authors are suggested to look into https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5602001/ and make appropriate modifications in the methods section including reporting of results. Thank you, this is an interesting suggestion. While we read carefully the paper entitled " How to Construct a Mixed Methods Research Design" [1], we believe we followed appropriate methods as described in [2][3][4][5][6][7][8] . . To develop and validate a new scale to evaluate the usability of any electronic healthcare system we followed the methodology as described in [2][3][4][5][6][7][8] as well as on the Imperial College London website (https://www.imperial.ac.uk/education-research/evaluation/tools-andresources-for-evaluation/questionnaires/how-a-scale-is-developed/). 2. This methodology has been used in the validation of different scales [8][9][10][11][12][13]. To make our process clearer, we added a few lines on the first paragraph of the "Methods" section, Page 9.Along with this the authors are also suggested to use the MMAT tool to check the completeness and quality of the final manuscript https://www.pcne.org/upload/wc2013/Lectures/Pluye.pdf Thank you for this suggestion. Through this paper we present a combination of quantitative and qualitative methods. The MMAT was designed for systematic mixed studies reviews [14]. We do recognise though that the MMAT can also be used to appraise the quality of empirical studies. Under the MMAT guidelines, our work is an exploratory design (QUALITATIVE proposal, followed by QUANTITATIVE generalization/tool development). The results are concluded by sequential synthesis: we analysed the qualitative study before the quantitative. We have added this clarification to the methods section.